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Abstract 


The  objective  of  this  study  was  to  determine  the 
usefulness  of  Earned  Value  Management  as  a  program 
management  tool  for  the  Department  of  Defense  acquisition 
community.  In  making  this  determination,  the  study  sought 
to  uncover  information  about  Earned  Value  Management  from 
the  contractor's  perspective  as  well  as  the  government 
administrator's  perspective.  It  also  sought  to  determine 
the  usefulness  of  Earned  Value  Management  during  the 
different  phases  of  the  software  acquisition  process. 

The  study  utilized  a  questionnaire  to  acquire  the  data 
necessary  for  analysis.  This  data  was  analyzed  to  compare 
perceptions  of  the  government  and  contractor  communities  in 
regard  to  the  use  of  Earned  Value  Management  as  a  program 
management  tool.  It  also  compared  perceptions  in  regard  to 
using  Earned  Value  Management  during  different  stages  of 
the  software  program  life  cycle. 
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implementation  of  eabned  value  management 

INTO  THE  SOFTNABE  ACQUISITION  PROCESS 


I .  Introduction 


Background 

The  Department  of  Defense  (DOD)  has  major  difficulties 

in  translating  mission  needs  into  efficiently  procured  and 

operational  systems.  One  of  the  aspects  of  this  difficulty 

is  the  acquisition  of  the  software  systems  that  comprise 

the  heart  and  soul  of  the  product.  In  the  absence  of  an 

efficient  and  effective  software  system,  the  overall 

acquisition  program  will  experience  skyrocketing  cost  and 

schedule  variances.  GAO  reports  on  nine  DOD  software 

contracts  cite  cost  overruns  as  fairly  common  in  more  than 

50  percent  of  the  cases  and  schedule  overruns  in  more  than 

60  percent  (Frank,  1983).  Of  $6.8  million  expended  on  the 

contracts,  the  results  included  the  following: 

Software  delivered,  but  never  used:  $3.2  million 

Software  paid  for,  but  never  delivered:  $1.9  million 

Software  extensively  reworked  before  use:  $1.3  million 
Software  used  after  changes:  $198,000 

Software  used  as  delivered:  $119,000 
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Effective  project  management  is  key  to  the  successful 
development  of  software  projects  in  an  overall  acquisition 
program.  According  to  the  Project  Management  Institute, 
project  management  is: 

The  art  of  directing  and  coordinating  human  and 
material  resources  throughout  the  life  of  a  project  by 
using  modern  management  techniques  to  achieve 
predetermined  objectives  of  scope,  cost,  time,  quality 
and  participant  satisfaction.  (Caudle  1999) 

Through  the  use  of  adequate  and  appropriate  project 

management  tools,  acquisition  officials  can  have  control  of 

the  software  development  process  by  guarding  against  cost, 

schedule  and  performance  deviations. 

In  the  face  of  reduced  budgets  and  increased 

congressional  scrutiny,  the  DOD  must  find  ways  to  acquire 

their  complex  and  unique  systems  more  efficiently  and 

effectively.  DOD  Directives  5000.1  and  5000. 2-R  set  the 

basis  for  accomplishing  acquisition  goals  in  this  new 

environment.  Under  the  category  of  organizing  for 

efficiency  and  effectiveness,  DODD  5000.1  recognizes  the 

need  for  limited  reporting  requirements,  automated 

acquisition  information  and  management  control. 

Limited  reporting  requirements  are  an  attempt  to 
prevent  the  government  from  overburdening  contractors  with 
requests  for  engineering  information  and  data  presentation. 
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which  the  government  cannot  fully  analyze  and  assess 
anyway.  Reporting  requirements  should  be  limited  to 
information  necessary  to  understand  program  status  and  make 
informed  decisions. 

Automated  acquisition  information  (AAI)  involves 
having  an  infrastructure  to  provide  current  and 
comprehensive  information  to  decision-makers.  It  attempts 
to  give  program  managers  access  to  management  information 
tools  that  facilitate  efficient  and  effective  acquisition. 
It  involves  having  readily  available  information  to 
encourage  responsiveness  to  the  internal  and  environmental 
factors  that  effect  the  dynamic  and  complex  software 
acquisition  process. 

Management  control  is  absolutely  necessary  to  promote 
effectiveness  and  accountability.  Achieving  efficiency  and 
effectiveness  means  maximizing  the  utilization  of  limited 
resources  through  management  control.  Cost,  schedule  and 
performance  parameters  must  be  understood,  quantified  and 
controlled  to  ensure  successful  project  acquisition. 
Deviations  from  Acquisition  Program  Baselines  (APBs)  must 
be  identified  and  countered  at  the  earliest  possible 
indication  to  prevent  cost,  schedule  or  performance 
variations  threatening  program  existence. 
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Problem  Statement 


Earned  Value  Management  is  a  project  management  tool 
available  to  the  acquisition  community  for  managing  these 
deviation  risks.  This  management  tool  attempts  to 
facilitate  and  satisfy  the  requirements  of  DODD  5000.1  for 
achieving  limited  reporting  requirements,  automated 
acquisition  information  and  management  control. 

Earned  value  is  used  to  break  the  software  project 
down  into  distinct,  identifiable  units  and  place  a  budgeted 
cost  and  schedule  allotment  against  each  of  these  units. 

The  system  is  designed  to  allow  immediate  access  into  the 
current  progress  of  work  completed  and  the  cost  and 
schedule  expended  in  achieving  this  work.  This  management 
program  seeks  to  identify  cost,  schedule  and  performance 
risks  as  early  as  possible  by  identifying  when  deviations 
occur  from  the  baseline  objectives.  This  allows  management 
to  direct  attention  to  the  problem  or  perform  trade-off 
analysis  to  keep  the  project  within  cost,  schedule  and 
performance  constraints. 

In  order  for  Earned  Value  Management  to  be  successful, 
there  are  several  requirements  that  the  management  system 
must  satisfy.  The  potential  of  Earned  Value  Management  must 
be  appreciated  and  realized  if  it  is  to  be  successfully 
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implemented  into  the  program  management  scheme.  The 
success  of  Earned  Value  Management  is  based  on  the 
following:  capability  over  prior  systems  of  management 
control,  contractor  and  government  usability,  insight  and 
representation  of  program  metrics,  adequacy  for  use  over 
the  life  cycle  of  the  project. 

The  fulfillment  of  these  objectives  is  necessary  to 
help  the  federal  government  bring  the  acquisition  of 
software  products  under  cost,  schedule  and  performance 
control.  The  applicability  and  feasibility  of  Earned  Value 
Management  needs  to  be  discerned  before  the  DOD  spends  time 
and  effort  instituting  this  program  management  scheme,  only 
to  find  it  to  be  unimproved  over  current  project  management 
methodologies . 

Research  Objectives 

The  objective  of  this  study  is  to  determine  and 
compare  government  and  contractor  perceptions  of  the 
usefulness  and  applicability  of  the  Earned  Value  Management 
philosophy.  The  ultimate  success  or  failure  of  the  earned 
value  philosophy  is  dependent  on  its  useful  application  by 
the  practicing  acquisition  community  —  both  by  government 
and  contractor  representatives.  These  perceptions  will 
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provide  information  useful  in  determining  the  legitimacy 
and  power  of  the  Earned  Value  Management  process. 

This  objective  is  sought  by  determining  the  level  of 
insight  acquisition  professionals  familiar  with  earned 
value  have  attained  and  desire.  The  degree  of  usability 
will  also  be  gauged.  This  should  help  determine  the 
effectiveness  of  Earned  Value  Management  in  the  fulfillment 
of  its  role  as  a  productive  program  management  tool. 

There  are  two  other  purposes  of  the  study.  The  study 
seeks  to  understand  the  applicability  of  earned  value 
across  the  entire  software  life  cycle.  It  is  sought  to 
determine  the  perceived  usefulness  of  earned  value  in 
comparison  to  previous  program  management  schemes.  These 
objectives  will  help  determine  whether  earned  value  is  a 
broad-based  application  or  an  augmentation  of  current 
program  management  tools,  applicable  during  only  specific 
portions  of  the  acquisition  process. 

Research  Focus 

This  study  will  focus  on  the  perceptions  of  the 
usefulness  and  applicability  of  the  Earned  Value  Management 
philosophy.  This  will  be  accomplished  with  analysis  of 
perceptions  from  both  from  the  contractor  and  the 
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government  perspective.  This  will  provide  insight  into  the 
quality  and  utility  of  the  Earned  Value  Management  process 
for  software  project  throughout  the  software  life  cycle 
from  each  sector's  point-of-view. 

This  study  also  is  an  analysis  of  the  comparison 
between  the  government  and  contractor  perspectives.  This 
will  be  useful  in  determining  whether  philosophy  gaps  exist 
and  if  the  government  is  adopting  commercial  or  directing 
government-mandated  practices . 

Furthermore,  this  study  seeks  to  report  the  types  of 
metrics  used  for  assessing  the  value  that  the  contractor 
has  earned  within  a  given  reporting  period.  This 
information  will  be  important  for  understanding  the 
standardization  and  specialization  that  exist  among  the 
broad  range  of  respondents.  This  information  may  also 
indicate  variation  in  the  application  and  utilization  of 
the  program  management  scheme. 

Methodology 

The  design  of  this  study  is  based  on  a  structured 
questionnaire  of  government  and  contractor  acquisition 
professionals.  The  questionnaires  were  distributed 
throughout  the  local  government  and  contractor  community 
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for  their  responses.  The  structured  questionnaire  consists 
of  statements  in  several  different  categories  and  their 
relation  to  the  Earned  Value  Management  process.  The 
categories  gauge  the  responses  based  on  the  Likert  Scale  by 
choosing  one  of  five  agreement  choices  (Cooper  1979)  . 

The  responses  will  be  distinguished  between  government 
and  contractor  subjects.  The  responses  will  be  analyzed  to 
determine  the  mean  level  assessment  of  Earned  Value 
Management  for  each  of  the  identified  categories.  This 
will  determine  the  overall  level  of  satisfaction  from  both 
the  government  and  contractor  perspective  of  the  success  of 
the  earned  value  scheme. 

The  categorized  responses  will  then  be  analyzed 
between  government  and  contractor  subjects  to  determine 
differences  in  the  perceptions  of  the  utility  of  Earned 
Value  Management.  This  analysis  of  means  will  help 
determine  whether  or  not  the  government  and  contractor 
realize  the  same  benefit  or  detriment  from  the  use  of  the 
earned  value  scheme. 

As sumptions /Limitations 

This  study  does  not  distinguish  between  the  size  and 
complexity  of  software  acquisition  projects,  which  may  be  a 
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determinant  of  the  project  management  scheme  employed.  It 
also  does  not  distinguish  between  the  type  of  software 
project,  which  may  range  from  space  command  and  control  to 
aircraft  navigation  to  accounting  and  finance  budgeting 
systems.  The  cost  estimation  and  work  breakdown  structure 
units  may  be  of  dissimilar  size  and  scope,  which  may  affect 
the  relative  utility  of  the  earned  value  success  in 
tracking  and  controlling  work  progression. 

The  only  distinction  for  the  subjects  of  this  study  is 
whether  the  participant  is  identified  as  a  government 
official  or  a  contractor.  This  does  not  take  into  account 
differences  between  senior,  mid  and  lower  level  managers. 
There  is  also  not  a  distinction  between  whether  the 
government  officials  are  military  or  government  civilians. 

Implications 

The  results  of  this  study  could  yield  important  and 
crucial  information  for  government  acquisition  executives. 
This  study  will  determine  the  perceived  usefulness  and 
applicability  of  Earned  Value  Management  in  the  current 
acquisition  environment.  It  will  help  in  determining  if 
philosophical  and  perceptual  differences  exist  between  the 
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contractor  and  government  communities  concerning  the 
'Utility  of  the  Earned  Value  Management  System. 

The  results  may  validate  a  program  manager's  decision 
to  employ  Earned  Value  Management  or  the  search  for 
alternative  program  management  schemes.  Analysis  of  the 
objectives  (capability,  usability,  insight  and  life  cycle 
adequacy)  may  suggest  that  the  program  manager  implement  an 
Earned  Value  Management  System,  find  an  alternative  system, 
or  retain  the  current  system. 

Preview 

The  control  of  cost,  schedule  and  performance  are 
paramount  to  the  success  of  a  software  acquisition. 
Identifying  deviations  from  the  baseline  at  the  earliest 
possible  instance  is  absolutely  necessary  to  alleviate 
risk,  development  alternatives,  and  budget  modifications  to 
allow  the  acquisition  to  continue.  Earned  Value  Management 
has  the  potential  to  identify  and  alleviate  the  risk  if 
properly  implemented.  The  key  to  the  effective  use  of 
earned  value  is  an  adequate  management  control  system  that 
fosters  the  proper  planning  and  integration  of  work  on  a 
project  (Christensen  and  Gordon  1998) .  This  study  seeks  to 
provide  factual  information  concerning  the  applicability  of 
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Earned  Value  Management  in  the  software  acquisition 
process . 
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I I .  Literature  Review 

This  chapter  is  intended  to  give  an  overview  of  some 
of  the  problems  that  have  been  inherent  in  the  software 
acquisition  process.  It  is  not  sought  to  find  a  resolution 
for  these  problems,  but  to  set  the  basis  for  why  Earned 
Value  Management  has  been  developed  and  problems  it 
attempts  to  overcome.  The  chapter  will  then  cover  the 
software  acquisition  life  cycle,  with  a  description  of  each 
of  the  phases.  It  will  differentiate  the  stages  of  the 
life  cycle  with  elements  that  are  peculiar  to  each  stage. 
The  chapter  will  conclude  with  a  comprehensive  discussion 
of  the  Earned  Value  Management  process.  This  discussion 
will  center  on  the  components  of  earned  value  and  how  they 
may  be  applied  to  the  stages  of  the  software  acquisition 
process. 

Software  Requirements  emd  Definition 

Problems  with  Translation  of  Requirements.  The 
translation  from  user  needs  into  a  viable  concept  involves 
the  interaction  of  several  groups.  The  primary  groups  are 
the  operations  and  acquisition  communities.  Economic 
perspectives  might  hinder  the  operations  community.  The 
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user  seeks  to  simplify  processes,  because  this  will 
increase  productivity  and  lower  operations  costs.  They 
desire  easily  understood  and  applied  capabilities,  which 
may  be  unreasonable  expectations  of  the  complex  software 
behind  the  system  (DeMarco,  1982) .  The  initial  requirements 
are  often  embryonic,  as  they  evolve  from  the  inception  of 
the  program  through  statement  of  work,  proposal  preparation 
and  development,  in  order  to  take  advantage  of  the  latest 
technological  breakthroughs  (Marciniak  and  Reifer,  1990) . 
Continuous  management  of  new  requirements  from  the  user  and 
derived  requirements  conceived  by  the  developer  is 
absolutely  necessary  because  of  their  impact  on  cost 
estimation . 

The  acquisition  community  is  often  hindered  by  lack  of 
practical  experience  with  the  systems  they  are  procuring. 
They  may  be  armed  with  training  and  book  knowledge  about 
the  system,  but  they  have  never  experienced  the  intricacies 
and  peculiarities  involved  with  operating  the  system  —  if 
they  have  ever  been  involved  in  operations  at  all.  They 
could  have  trouble  understanding  the  basis  or  reasoning 
behind  detailed  user  requirements;  thus  they  should  spend 
extensive  time  reviewing  and  verifying  the  requirement  to 
ensure  completeness,  correctness  and  relevance  (Cooper  and 
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Fisher,  1979) .  They  have  to  work  closely  with  the 
operations  group,  a  resource  adept  in  the  situation 
experienced  with  the  current  system.  Elaborate  design  and 
interfaces  are  useless  if  the  users  reject  them,  perceiving 
the  system  does  not  meet  their  needs  (Cooper  and  Fisher, 
1979). 

Estimation  of  Software  Projects  Difficult.  Estimation 
can  be  a  particularly  difficult  task  for  the  complex  and 
revolutionary  systems  that  are  typical  of  the  DOD.  Many  of 
these  custom  systems  and  applications  do  not  have  any 
standards  to  base  an  estimate  from,  especially  when 
considering  the  software  system  may  be  for  a  developmental 
aircraft  or  space  command  and  control  system.  Lack  of 
estimating  expertise,  biases,  inflated  expectations,  a  poor 
understanding  of  the  development,  political  considerations, 
and  failure  to  estimate  based  on  past  performance  are 
problems  experienced  in  estimation  (DeMarco  1982) . 

The  DOD  also  hinders  the  estimation  process.  Vague  or 
incomplete  specifications  limit  the  ability  to  interpret 
and  properly  apply  the  amount  work  to  be  accomplished  to 
the  cost  estimation  model.  This  leaves  to  the  estimator's 
discretion  the  scope  and  size  of  the  job.  The 
specifications  may  fail  to  state  directly  what  is  needed. 
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but  instead  specify  the  user's  solution  to  the  problem. 

This  restricts  the  creativity  of  the  developer,  disallowing 
a  more  cost  and  productivity  efficient  solution  (Shamlin 
1985) .  The  estimate  also  may  not  be  independent  and 
objective,  but  subject  to  estimator  project  involvement, 
competitive  business  practice  or  program  manager  influence 
(DeMarco  1982) .  These  factors  will  likely  provide  an 
inadequate  estimation,  setting  the  basis  that  may  be 
impossible  for  the  program  manager  to  attain. 

The  conversion  of  the  specifications  into  proper  cost 
estimation  determinants  must  also  be  done  in  a  manner  to 
properly  size  and  fund  the  project.  The  value  of  the  cost 
estimation  model  is  only  as  good  as  the  inputs  and  their 
representation  of  the  work  that  must  be  accomplished.  A 
standard  used  for  cost  estimation  is  Lines  of  Code  (LOG) . 
The  difficulty  in  using  LOG  for  software  estimation  is  that 
the  number  needed  is  not  well  known  until  the  coding  is 
near  completion,  while  the  estimate  must  be  made  before  the 
requirements  analysis  for  budgeting  and  planning  purposes. 

There  are  other  problems  inherent  in  using  LOG  as  a 
basis  for  estimate.  There  are  no  national  or  international 
standards  for  the  application  of  LOG  that  encompasses  all 
procedural  languages.  LOG  may  be  counted  using  physical  or 


15 


logical  lines.  LOG  counts  may  include  executable  lines, 
data  type  definition,  comments  and  even  blank  lines. 
Reusable  code  may  be  counted  several  times.  Code  reuse 
accounts  for  20-30  percent  of  programming  in  procedural 
languages  such  as  C,  COBOL  or  FORTRAN  (Jones  1991) . 

Software  may  be  developed  using  spreadsheets, 
program  generators  or  embedded  commercial-off-the-shelf 
(COTS)  programs,  where  the  development  of  LOC  is  less 
significant.  The  type  of  language  being  utilized  will  have 
an  impact  on  the  LOC  level,  with  the  higher  level  languages 
providing  LOC  estimates  that  are  incomparable  with  code¬ 
intensive  lower  level  languages.  Some  applications  employ 
the  use  of  several  different  languages,  such  as  Ada  mixed 
with  assembler  language  (Jones  1991) . 

Translation  of  Estimation  into  Effective  Program 
Management  Measures.  This  LOC  estimate  does  not  take  into 
account  several  of  the  non-coding  factors  that  have  major 
implications  on  the  cost  and  schedule  of  the  acquisition. 
These  are  termed  hard  and  soft  data.  The  hard  and  soft 
data  must  be  matched  with  the  LOC  normalized  data  to 
provide  realistic  estimates  of  the  project. 

Hard  data  includes  the  number  of  members  to  be 
assigned  to  the  project.  When  determining  this  number,  the 
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amount  of  time  allocated  to  each  of  the  project  tasks,  such 
as  requirement  definition  and  product  design,  must  be 
distinguished  from  programming  time.  The  schedule  duration 
of  project  tasks  is  included,  accounting  for  the  overlap 
and  concurrency  of  tasks  being  performed  in  parallel.  The 
amount  of  documentation  and  test  cases  to  be  required  is 
involved  in  the  estimation.  The  number  of  bugs  and  defects 
expected  are  also  budgeted.  These  hard  data  items,  with 
little  or  no  subjectivity,  are  more  easily  quantifiable 
than  soft  data  items  (Jones  1991) . 

Soft  data  is  more  difficult  to  measure  and  budget, 
because  much  of  it  is  based  on  human  opinion.  The  skill 
and  experience  of  the  team  is  a  major  consideration.  The 
make-up  of  the  team  will  likely  range  from  recent  college 
graduates  to  those  with  operational  experience  to  seasoned 
business  professionals.  Constraints  and  schedule  pressures 
may  come  from  within  the  development  team  or  from  the  user 
community,  who  needs  the  product  yesterday.  The  stability 
of  the  project  requirements  has  an  impact,  and  it  must  be 
controlled  by  the  program  manager  to  prevent  creeping 
requirements  from  steering  the  program  off  course.  Other 
determinants  include  expertise  and  cooperation  of  the  user. 


17 


tool  and  methodology  adequacy,  organizational  structure  and 
adequacy  of  office  space  (Jones  1991) . 

The  program  manager  must  adequately  allocate  the 
resources  made  available  from  the  budgetary  allocations  and 
authorizations.  Cost  and  schedule  are  constraints  on  the 
program  manager  in  achieving  the  level  of  performance 
necessary  to  complete  the  project.  A  translation  is  made 
between  the  LOC  or  function  points  basis  used  for  the  cost 
estimation  to  a  level  of  effort  (LOE)  parameter.  This  LOE 
is  a  resource  allocation  metric  that  will  measure  the 
amount  of  man-hours  that  have  been  expended  in  planning, 
programming  and  analysis  by  the  development  team.  A  major 
drawback  of  this  performance  measure  is  that  is  does  not 
measure  the  amount  of  progress  that  has  been  made  toward 
requirements  analysis,  program  design,  programming,  test 
and  integration. 

Software  Project  Development 

Software  System  Concept /Requirements  Analysis  Phase. 

The  systems  concept  phase  is  the  determination  of  the  need 
and  scope  of  the  software  system  to  be  acquired.  The  types 
of  systems  may  involve  a  software  intensive  (command  and 
control,  management  information  or  radar  interpretation). 
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software  reliant  (aircraft  avionics  or  missile  targeting) 


or  embedded  commercial-off-the-shelf  (COTS)  systems.  In 
either  case,  it  is  absolutely  necessary  that  the 
acquisition  community  have  a  clear  and  comprehensive 
understanding  of  the  users'  needs.  A  survey  of  over  8000 
software  projects  found  that  lac)<  of  user  input,  incomplete 
requirements  and  changing  requirements  were  the  top  reasons 
that  projects  were  delivered  late,  over  budget  and  with 
less  functionality  than  desired  (The  Standish  Group,  1994) . 

The  foundation  of  the  software  project  is  based  on 
four  documents.  These  documents  include  the  Concept  of 
Operations  (CONOPS) ,  System  Requirements  Specifications, 
Software  Requirements  Specifications  and  the  Project  Plan 
(Turner  1997) .  The  understanding  of  user  need  is  heavily 
reliant  on  the  validity  and  stability  of  these  documents 
and  their  incorporation  into  the  government-provided 
Statement  of  Objectives  (SOO) .  The  System  Requirements 
Review  (SRR)  and  System  Design  Review  (SDR)  are 
opportunities  for  the  validation  and  clarification  of  the 
requirements.  This  understanding  will  facilitate  the 
proper  estimation  and  determination  of  the  computer 
software  configuration  items  (CSCIs)  necessary  for  the 
software  acquisition  project  (Murtagh  1992) . 
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Planning  Phase.  The  planning  phase  involves  placing 
the  proper  personnel  and  material  in  a  manner  to  facilitate 
performance  while  minimizing  risk.  Planning  cannot  be 
effectively  accomplished  until  the  requirements  are  stable 
and  understood.  Internal  (e.g.  operational)  and  external 
(e.g.  Congressional)  pressures,  along  with  the  amount  of 
developmental  work  necessary,  may  affect  the  development  of 
milestone  schedules.  To  properly  utilize  resources,  the 
developers  should  have  a  detailed  Work  Breakdown  Structure 
(WBS)  to  place  the  resources  against.  A  network  schedule 
will  also  be  developed  to  determine  the  critical  path  and 
allow  resource  balance.  This  will  allow  the  allocation  of 
risk  and  resources  to  meet  schedule  requirements  while 
preventing  the  development  of  additional  critical  paths. 

Software  System  Design/Codinq  Phase.  Before  the 
software  system  design  phase  can  be  undertaken,  it  is 
absolutely  crucial  that  the  software  requirements  analysis 
was  done  well  (Murtagh  1992).  Often  the  software  community 
will  undertake  the  design  and  coding  before  the 
requirements  are  stable  and  understood,  which  eventually 
leads  to  extensive  rework  resulting  in  cost  and  schedule 
impacts.  Schedule  pressures  that  drove  the  premature 
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design  and  coding  ultimately  lead  to  waste  of  valuable 
resources  and  delays. 

The  primary  components  of  the  design  phase  are  the 
preliminary  and  detailed  design  reviews.  The  preliminary 
design  breaks  the  CSCIs  into  computer  software  components 
(CSCs)  and  determines  what  requirements  each  will  fulfill. 
The  interfaces  for  each  of  the  CSCs  are  identified  and 
documented.  All  the  requirements  from  the  SCO  and  the 
resulting  Statement  of  Work  (SOW)  are  addressed,  providing 
traceability  of  requirements  (Murtagh  1992) .  This  review 
is  a  combined  effort  of  the  developers,  customers  and 
users.  Any  changes  to  the  software  specifications  after 
the  PDR  should  be  reserved  for  the  next  system  upgrade 
(Turner  1997) . 

The  detailed  design  identifies  the  resources  that  will 
be  used  to  develop  each  CSC  by  breaking  them  dowts  into 
computer  software  units  (CSUs) .  According  to  DoD-STD- 
2167A,  this  involves  the  definition  of  the  structure, 
interfaces,  data  flow  and  control  flows  of  the  CSCs  and 
CSUs.  Although  not  the  current  standard,  the  ideas  are 
still  valid.  The  design  requirements  are  established  and 
finalized.  This  review  bridges  the  gap  between  the  top- 
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level  design  and  the  coding.  This  design  includes  a 

description  of  the  following  (Turner  1997) : 

software  components  (units  or  modules) 
interfaces  between  components 

-  data  structures 
input/output  screens 

-  timing  and  memory  constraints 

-  performance  measurements 

-  error  handling 

The  efforts  of  the  detailed  design  phase  culminate  in  the 
Critical  Design  Review  (CDR) .  Following  the  CDR,  coding 
and  unit  test  is  conducted  for  each  CSU. 

System  Implementation/Testinq  Phase.  The  software 
system  implementation  and  testing  phase  is  initiated  with 
the  reconstruction  of  the  tested  CSUs  into  CSCs.  The 
aggregate  CSCs  are  tested  with  the  integration  of  each  CSU. 
Testing  is  also  conducted  between  dependent  CSCs.  The  same 
testing  is  conducted  for  COTS  products  that  are  integrated 
into  the  software  system. 

During  this  testing,  configuration  verification  and 
performance  validation  is  accomplished.  Not  only  do  the 
software  components  have  to  work  in  concert,  but  they  also 
have  to  provide  the  desired  functionality.  Verification 
ensures  the  CSCI  performance  meets  the  design 
specifications,  while  validation  ensures  that  the  software 
satisfies  the  intended  use.  The  verification  and 
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validation  team  should  be  independent  of  the  authority  of 
the  development  manager.  The  team  must  accomplish  several 
tasks  in  exercising  the  software,  including  the  following: 

-  traceability  analysis 
requirements  validation 

-  interface  analysis 

-  software  system  test  plan  generation 

-  acceptance  test  plan  generation 
algorithm  analysis 

-  auditing 

“  control  flow  analysis 

-  database  analysis 
simulation 

inspections /walk  through 

The  ultimate  purpose  of  testing  is  to  exercise  the  code  in 
order  to  expose  errors  (Turner  1997) . 

Operations  and  Maintenance  Phase.  The  operations  and 
maintenance  phase  should  be  a  relatively  stable  period  if 
the  requirement  analysis,  design,  testing  and  integration 
were  thorough  and  successful.  There  are  several  other 
factors  that  must  be  considered,  though.  Dependent  on  the 
type  of  project,  it  might  be  too  cost  and  schedule 
extensive  to  test  all  the  bugs  out  of  the  software.  There 
are  often  fixes  that  are  required  after  fielding  the 
system.  Specification  changes  introduced  after  the  PDR 
will  need  to  be  incorporated  into  system  releases  and 
upgrades.  Operational  requirement  changes  will  necessitate 
coding  modification.  Schedule  constraints  may  also  drive 
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the  need  for  pre-planned  product  improvement  (P^I)  measures, 
providing  the  functionality  that  would  have  been  available 
if  adequate  time  had  been  given  to  fully  design  and  test 
the  system. 

Earned  Value  Management 

Overview.  Earned  Value  Management  (EVM)  is  a  system 
that  can  help  identify  and  prevent  the  shortcomings 
associated  with  managing  by  LOE  tracking.  EVM  is  a 
technique  that,  coupled  with  an  effective  cost  estimation 
method,  provides  the  program  manager  with  valuable 
information  about  how  a  program  is  progressing  at  any  given 
time.  EVM  uses  the  same  work  units  used  in  the  cost 
estimation;  it  applies  a  dollar  and  time  value  to  each  of 
the  work  units.  No  longer  does  the  manager  have  to  simply 
manage  based  on  cost  and  scheduled  manpower,  which  tells 
nothing  of  the  actual  progress  that  has  been  accomplished. 

The  earned  value  concept  has  existed  within  the 
government  for  decades  in  an  inflexible  and  formalized 
manner.  The  earned  value  idea  was  initially  mandated  with 
the  imposition  of  35  Cost/Schedule  Control  Systems  Criteria 
(C/SCSC) .  All  firms  wishing  to  participate  in  cost 
reimbursable  or  incentive  contracts  were  required  to  adhere 
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to  the  criteria,  which  some  considered  to  be  an  Utopian 
ideal  that  was  difficult  to  emulate  and  employ  on  all 
contracts  (Fleming  and  Koppelman  1998) .  Although  C/SCSC 
provided  useful  information  in  predicting  cost  and  schedule 
results,  industry  resisted  C/SCSC  because  it  contained  too 
many  perceived  non-value  added  requirements. 

Earned  Value  Management  in  its  current  form  is  a  more 
cooperative  program  between  the  government  and  the 
industry.  After  years  of  imposing  strict,  mandated 
criteria  on  the  contractors,  the  government  encouraged 
feedback  and  incorporated  it  into  the  requirements  of  the 
program  management  scheme  to  be  developed.  The  National 
Security  Industry  Association  (NSIA)  was  chartered  with 
assessing  the  utility  of  the  earned  value  criteria  and 
rewording  the  criteria  to  make  it  more  palatable  to  the 
project  management  community  (Fleming  and  Koppelman  1998*. 
The  current  environment  even  allows  tailoring  to  meet  the 
needs  of  the  contractor,  especially  since  it  a  contractor- 
owned  instead  government-imposed  system  (evms.dcmdw.dla.mil 
1998) .  The  tailored  Earned  Value  Management  System  (EVMS) 
must  simply  be  certified  and  accepted  by  the  government . 

EVM  in  the  Software  Environment.  The  results  of  the 
cost  estimation  must  put  the  work  to  be  done  in  measurable 
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units  that  can  be  tracked  using  EVM.  The  government 
includes  EVM  as  the  cost  and  schedule  tracking  system  in 
the  Statement  of  Work  and  contracts  for  it  accordingly. 
There  are  several  different  work  unit  groupings  that  could 
be  employed:  product,  component,  activity,  defect, 
enhancement,  phase,  LOC  or  objects  (Grady  1992) .  The  work 
unit  measurements  may  also  be  applied  to  individual  work 
teams  (Harvey  1995) .  The  program  manager  should  allow  the 
contractor  to  determine  the  appropriate  measures  that  will 
best  fit  the  contractor's  recording  methods,  since  the 
actual  monitoring  and  reporting  is  accomplished  by  the 
contractor.  The  status  is  reported  at  regularly  scheduled 
status  meetings,  on  demand  or  through  access  via  Electronic 
Data  Interchange  (EDI) . 

Application  of  EVM.  EVM  is  comprised  of  three 
components.  Unlike  previous  methods  of  tracking,  EVM 
monitors  the  number  of  work  units  (not  dollars  or  man¬ 
hours)  versus  time.  Budgeted  Cost  of  Work  Scheduled  (BCWS) 
is  the  number  of  work  units  scheduled  for  completion  at  a 
particular  time  based  on  the  results  of  the  cost 
estimation.  Actual  Cost  of  Work  Performed  (ACWP)  is  the 
number  of  work  units  that  have  been  completed  at  a 
P^^ticular  juncture  in  a  program.  BCWP  shows  how  many  work 
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units  have  been  completed  in  comparison  with  the  amount 
estimated  to  be  complete  (BCWS)  and  the  amount  actually 
used  (ACWP)  to  get  to  the  present  (Cruver  1997) .  Following 
is  an  example  of  EVM  in  practice: 


Figure  1.  EVM  in  Application 


At  point  number  9  in  Figure  1,  15  work  units  were  scheduled 
for  completion.  Only  10  work  units  had  actually  been 
completed  at  this  point,  but  it  took  17  of  the  estimation- 
based  units  to  complete  the  work  that  had  been  done. 

The  cost  and  schedule  variance  can  easily  be 
determined  from  the  preceding  figure.  The  variance 
equations  are  as  follows : 


CV  =  BCWP 

-  ACWP 

(1) 

SV  =  BCWP 

-  BCWS 

(2) 

where  CV  is  cost  variance,  BCWP  is  Budgeted  Cost  for  Work 
Performed,  ACWP  is  Actual  Cost  for  Work  Performed  and  BCWS 
is  Budgeted  Cost  for  Work  Scheduled. 


The  cost  variance  is  determined  by  subtracting  ACWP  from 
BCWP .  This  determines  the  difference  in  the  number  of  work 
units  that  were  completed  and  the  actual  number  of  work 
units  expended  for  the  work  done.  The  program  manager  got 
10  work  units  for  the  price  of  17  work  units,  meaning  there 
is  a  cost  variance  of  -7  work  units. 

The  schedule  variance  is  determined  by  subtracting  the 
BCWS  from  BCWP.  The  estimation  determined  15  work  units 
should  be  completed  at  this  point,  but  only  10  were.  The 
program  manager  now  has  a  schedule  variance  of  -5  work 
units.  The  program  manager  can  easily  discern  at  this 
point  that  actual  work  completed  is  behind  schedule  and 
costing  much  more  than  had  been  budgeted. 

Estimate  at  Completion  (EAC)  is  the  total  amount  of 
work  units  from  the  estimation  model  for  accomplishing  the 
software  acquisition  project.  This  number  is  the  baseline, 
but  will  likely  be  adjusted  based  on  actual  results. 

There  are  several  methods  of  determining  an  EAC.  The 
Budget  at  Completion  (BAC)  is  the  budget  based  on  the 
original  estimation  for  the  contract  performance.  The  Cost 
Performance  Index  (CPI),  which  is  the  ratio  of  budgeted 
cost  for  work  performed  to  the  actual  cost  incurred,  is 
determined  by  the  following: 
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CPI  =  BCWP 
ACWP 


(3) 


The  first  method  of  determining  EAC  assumes  that  the  cost 

performance  experienced  to  date  will  remain  the  same  for 

the  remainder  of  the  contract. 

Method  1:  EAC  =  (4) 

CPI 

The  next  method  of  determining  EAC  assumes  that  past 
performance  is  not  indicative  of  future  performance  on  the 
contract.  This  method  does  allow  for  the  revision  of  the 
planned  cost  based  on  the  historical  data  from  completed 
work  packages. 

Method  2:  EAC  =  OCW  +  RCWNYB  (5) 

CPI 

where  OCW  is  Open  and  Completed  Work  and  ARCWNYB  is  Revised 
Cost  of  Work  Not  Yet  Begun. 

Another  method  of  determining  EAC  assumes  that  the 
remaining  work  is  independent  of  work  completed  to  date. 
This  may  be  unrealistic  as  work  packages  may  influence 
performance  of  subsequent  work  packages— especially  on  the 
critical  path.  This  method  is  very  risky  if  revision  of 
the  estimates  is  not  allowed. 

Method  3:  EAC  =  ACWP  +  RWPC  (6) 
where  RWPC  is  Remaining  Work  at  Planned  Cost. 
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There  are  several  other  methods  available  for  the 
determination  of  EAC,  with  their  value  differing  based  on 
the  dollar  value,  risk,  cost  accounting  system  and 
estimation  accuracy  of  the  project  (Kerzner  1998:815-6). 

As  soon  EAC  begins  to  rise  above  baseline,  the  program 
manager  should  find  ways  to  cut  costs  or  secure  additional 
funding.  At  this  time,  the  program  manager  should  confer 
with  the  contractor  to  determine  the  Latest  Revised 
Estimate  (LRE) ,  which  is  the  best  guess  concerning  what  it 
will  cost  in  work  units  to  complete  the  program  (Cruver 
1997) .  This  is  a  crucial  decision  point  to  determine 
whether  or  not  to  continue  the  program,  as  errors  and  poor 
judgment  may  lead  to  escalation  and  "throwing  good  money 
after  bad"  (Kiel  1995) . 
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III.  Methodology 


This  chapter  describes  the  methodology  used  in 
acquiring  and  analyzing  the  data  used  in  the  conduct  of  the 
study.  It  will  describe  the  data  collection  instrument 
development,  the  administration  of  the  data  collection 
instrument,  the  subjects  and  the  analysis  to  be  performed 
on  the  gathered  information. 

Justification  of  Approach 

A  structured  questionnaire  is  used  to  collect  the  data 
for  analysis  due  to  the  fact  that  data  concerning  user 
satisfaction  with  EVM  is  not  currently  available.  The 
questionnaire,  with  room  for  comments  on  each  question, 
will  allow  the  participants  to  openly  and  candidly  comment 
on  each  of  the  statements  anonymously.  A  questionnaire 
will  also  allow  the  participants  to  answer  at  their 
discretion  within  time  limitations  necessary  for  adequate 
analysis,  as  many  acquisition  officials  cannot  set  aside  a 
specific  for  these  purposes.  A  preferred  approach  would  be 
to  conduct  personal  interviews  to  record  subjects'  tone  and 
gestures,  but  this  would  be  at  the  expense  of  a  great  deal 
of  time,  coordination  and  respondent  anonymity. 
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Instrument  Development 


The  design  of  this  study  is  based  on  a  structured 
questionnaire  consisting  of  25  statements  in  regard  to 
perceptions  of  several  categories  (insight,  usability, 
etc.)  and  their  relation  to  the  Earned  Value  Management 
process.  Another  question  regards  performance  metrics 
utilized  for  tracking  cost,  schedule  and  performance.  The 
entire  questionnaire  is  included  as  Appendix  E. 

The  questionnaire  is  limited  to  25  statements  and  one 

) 

question  to  keep  it  from  requiring  too  much  of  the 

respondent's  time.  It  is  hoped  that  this  will  keep  the 

respondent's  attention,  allow  for  honest  and  thoughtful 

answers  and  improve  the  response  rate.  The  response 

pattern  is  based  on  the  Likert  Scale,  with  responses  to 

each  statement  limited  to  the  choice  of  one  of  five 

possible  agreement  levels  (Cooper  1979) .  There  will  be  an 

area  following  each  question  to  allow  for  comments,  which 

could  provide  useful  insight  into  the  response.  The 

response  scale  range  follows: 

5  -  Strongly  Agree 
4  -  Agree 

3  -  Neither  Agree  or  Disagree 
2  -  Disagree 
1  -  Strongly  Disagree 
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Four  broad  categories  set  the  basis  for  the  analysis 

of  user  satisfaction  and  utility  in  the  administration  of 

the  Earned  Value  Management  System.  The  categories  (with 

their  corresponding  appendix)  include  the  following: 

Insight/Representation  (A) 

Usability  (B) 

Comparison  with  previous /other  current  systems  (C) 

Life  cycle  applicability  (D) 

Insight  and  Representation.  The  insight  and 
representation  statements  are  used  to  determine  the 
satisfaction  of  acquisition  community  members  with  the 
ability  of  Earned  Value  Management  to  provide  precise, 
meaningful  information.  Insight  pertains  to  the 
availability  of  programmatic  information  in  the  proper 
amount  and  depth  necessary  to  understand  program  status  and 
make  informed  decisions.  Representation  deals  with  work 
depiction  accuracy  and  comprehension  of  the  portrayal  of 
the  contractor's  work  expenditures  and  accomplishments  to 
the  government  representative. 

There  are  ten  statements  regarding  insight  and 
representation.  The  statements  deal  with  precision, 
forecasting  utilization  and  translation  from  cost 
estimation  measures  into  identifiable,  representative  work 
units.  The  statements  deal  with  the  ability  of  the 
government  to  use  these  work  units  to  understand  and  budget 
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cost  and  schedule  progression  and  identify  variation  using 
Earned  Value  Management.  The  insight  questions  can  be 
referenced  in  Appendix  A. 

Usability.  Usability  pertains  to  the  complexity 
involved  in  the  utilization  of  the  Earned  Value  Management 
System  for  software  project  management.  The  usability 
statements  seek  to  determine  both  the  ease  of  use  of  the 
system  for  cost,  schedule  and  performance  tracking  as  well 
as  the  burden  shared  by  the  government  and  contractor  in 
using  the  Earned  Value  Management  System. 

There  are  six  statements  regarding  the  usability  of 
the  Earned  Value  Management  System.  The  statements  deal 
with  the  relative  ease  of  program  management  using  earned 
value,  understanding  of  the  earned  value  system  and 
government-contractor  obligations  to  the  system.  The 
questionnaire  also  contains  a  statement  regarding  the 
effectiveness  of  earned  value  for  hardware  systems 
acquisition  for  comparison  with  software  systems.  The 
usability  questions  can  be  referenced  in  Appendix  B. 

Comparison  with  Previous/Other  Current  Systems.  This 
category  is  being  used  to  determine  the  preference  for 
Earned  Value  Management  or  another  management  system  for 
software  program  management.  This  information  is  important 
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to  understand  government  and  contractor  preference  for 
either  system  based  on  capability  and  performance. 
Preference  for  a  previous  or  other  current  system  may  be 
indicative  of  a  natural  resistance  to  the  Earned  Value 
Management  philosophy. 

There  are  three  statements  with  regard  to  comparison 
with  previous/other  current  systems.  These  seek  to 
determine  the  adequacy  of  previous  systems,  whether  better 
program  management  systems  exist  and  whether  the  subject 
perceives  Earned  Value  Management  to  be  just  a  current 
buzzword.  These  statements  will  help  determine  whether 
Earned  Value  Management  lives  up  to  its  billing  in 
providing  cost  and  schedule  management  capability  not 
otherwise  available.  The  comparability  questions  can  be 
referenced  in  Appendix  C. 

Life  Cycle  Applicability.  Life  cycle  applicability 
statements  pertain  to  the  application  of  the  Earned  Value 
Management  System  across  all  stages  of  the  software 
development  life  cycle.  This  information  will  help  in 
understanding  whether  the  acquisition  community  feels  this 
program  management  system  is  a  stand-alone  tool,  or  one  of 
multiple  tools  program  managers  need  to  have  at  their 
disposal.  This  will  also  be  useful  in  determining  the 
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appropriateness  of  earned  value  for  project  management,  as 
complexities  and  information  loss  are  risks  associated  in 
the  transition  between  different  management  systems  at 
successive  stages  of  the  life  cycle. 

There  are  six  statements  regarding  life  cycle  process 
applicability.  These  statements  determine  whether  the 
system  concept /requirements  analysis,  planning, 
design/coding,  implementation/testing  and 
operation/maintenance  phases  are  enhanced  by  the  use  of 
Earned  Value  Management  as  the  program  management  tool. 

The  life  cycle  applicability  questions  can  be  referenced  in 
Appendix  D. 

Instrtunent  Administration 

The  questionnaire  was  distributed  through  mailings  and 
electronic  mail  to  the  various  segments  of  the  government 
acquisition  community.  It  was  also  distributed  to  the 
contractor  community  through  meetings  of  the  Dayton  chapter 
of  the  Institute  of  Electrical  and  Electronics  Engineers 
(IEEE)  and  Association  for  Computing  Machinery  (ACM) .  The 
questionnaires  were  taken  at  the  respondent's  leisure 
within  deadlines  necessary  for  adequate  analysis.  The 
questionnaires  were  accompanied  with  a  self-addressed. 
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stamped  envelope  (SASE)  for  collection.  The  SASE  is 
included  for  respondent  convenience  and  to  encourage 
enhanced  return  rates. 

Data  Analysis 

The  responses  were  distinguished  between  government 
and  contractor  subjects.  The  responses  are  distinguished 
by  identification  as  a  government  or  non- government 
employee  at  the  end  of  the  questionnaire.  The  responses 
have  been  analyzed  to  determine  the  mean  level  assessment 
of  Earned  Value  Management  for  each  of  the  identified 
categories.  This  will  determine  the  overall  level  of 
satisfaction  from  both  the  government  and  contractor 
perspective  of  the  success  of  the  earned  value  scheme. 

The  categorized  response  levels  were  determined  by 
calculating  the  mean  score  for  the  questions  making  up  the 
category.  For  instance,  the  response  level  for  the 
category  of  insight  and  representation  was  determined  by 
the  following: 

Meani  =  Zi...n  (A  +  B  +  G  +  P  +  Q  +  RtS+V-fX  +  Y) 

_ _ 10 _  (7) 

n 

A,  B,  etc.  =  Corresponding  statement  number  in  alphabet 
n  =  Number  of  respondents 
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The  corresponding  values  for  the  response  to  the  statements 
will  be  as  follows: 

5  -  Strongly  Agree 

4  -  Agree 

3  -  Neither  Agree  or  Disagree 

2  -  Disagree 

1  -  Strongly  Disagree 

The  categorized  responses  will  then  be  analyzed 
between  the  government  and  contractor  to  determine 
differences  in  the  perceptions  of  the  utility  of  Earned 
Value  Management.  This  analysis  of  means  will  help 
determine  whether  or  not  the  government  and  contractor 
realize  the  same  benefit  or  detriment  from  the  use  of  the 
earned  value  scheme. 

Conclusion 

This  section  describes  the  formulation,  administration 
and  analysis  of  the  study.  The  analysis  will  provide 
insight  into  the  perceived  applicability  and  utility  of 
Earned  Value  Management  according  to  those  using  it  in  the 
acquisition  community.  This  seeks  to  provide  evidence  as 
to  the  usefulness  or  confusion  resulting  from  the  broad 
application  of  the  Earned  Value  Management  System. 
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IV.  Data  Description  and  Analysis 


Overview 

The  structured  questionnaire  gauged  the  level  of 
government  and  contractor  responses  to  categories  including 
insight,  usability,  comparability,  and  life  cycle 
applicability  of  the  earned  value  philosophy.  The  response 
levels  are  presented  from  the  government,  contractor  and 
aggregate  perspectives.  Only  the  individual  statements 
that  deviated  from  the  category  mean  are  indicated  and 
discussed.  Statements  that  deviated  substantially  from  the 
mean  represent  areas  of  concern  that  potentially  need  to  be 
addressed  by  project  managers.  The  substance  of  the 
statements  that  are  not  individually  addressed  is  captured 
in  the  category  discussion. 

Insight 

The  respondents  indicated  that  EVM  allowed  adequate 
insight  into  project  management.  The  government 
respondents  indicated  a  higher  level  of  insight  using  EVM 
than  the  contractors.  The  questions  regarding  insight 
(Appendix  A)  had  an  overall  mean  response  level  of  3.425,  a 
government  mean  of  3.533  and  a  contractor  mean  of  3.317. 
This  indicates  that  government  employees  perceive  a  greater 
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level  of  project  management  insight  from  the  application  of 
EVM  than  contractors,  who  use  EVM  to  report  progress. 


Table  1 .  Means  for  Insight  Category  and  Questions 


Combined 

Government 

Contractor 

Overall  Insight 

3.425 

3.533 

3.317 

Adequacy  of  EVM  for 
tracking  cost  and  schedule 

4.083 

4.000 

4.167 

Ability  of  EVM  to  forecast 
cost  and  schedule  variance 

3.833 

4.000 

3.667 

Translation  from  cost  estimation 
to  EVM  tracking  parameters 

2.833 

2.667 

3.000 

Adequacy  of  current  software 
metrics  for  EVM  tracking 

3.083 

3.833 

2.333 

The  questions  that  deviated  substantially  from  the 
mean  will  now  be  addressed.  Respondents  indicated  a  high 
level  of  agreement  concerning  the  adequacy  of  earned  value 
information  in  tracking  cost  and  schedule  performance.  The 
mean  for  this  question  was  4.083,  with  the  government  and 
contractor  means  being  4.000  and  4.167,  respectively. 

There  were  also  high  response  levels  for  the  ability  EVM  to 
forecast  cost  and  schedule  variance,  with  the  aggregate 
mean  being  3.833,  government  4.000  and  contractor  3.667. 
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Respondents  were  less  enthusiastic  about  the  adequacy 
of  tools  for  translating  cost  estimation  parameters  into 
EVM  data.  Insight  can  be  inhibited  by  the  poor  conversion 
from  cost  estimation  into  program  management  tracking  data. 
The  mean  concerning  cost  estimation  translation  was  2.833, 
with  a  government  mean  of  2.667  and  a  contractor  mean  of 
3.000.  Respondents  also  indicated  that  current  software 
metrics  were  inadequate  for  tracking  progress  using  EVM. 

The  aggregate,  government  and  contractors  means  were  3.083, 
3.833  and  2.333,  respectively.  The  high  variability 
between  government  and  contractor  means  indicates  a 
disparity  in  the  usefulness  of  current  metrics  for  tracking 
progress . 

Usability 

Respondents  indicated  EVM  has  a  high  degree  of 
usability  in  project  management.  Both  the  government  and 
contractor  respondents  were  comfortable  with  the  usability 
of  EVM  for  effective  project  management.  The  questions 
regarding  usability  (Appendix  B)  had  an  aggregate  response 
level  of  3.889,  similarly  high  government  (4.083)  and 
contractor  (3.694)  means.  This  indicates  a  mutual 
understanding  of  the  use  and  application  of  EVM  for  project 
management . 
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Some  areas  stood  out  from  the  overall  category  mean. 
Respondents  gave  strong  indications  that  the  implementation 
of  EVM  has  not  been  time  or  cost  prohibitive.  This 
indicates  that  the  training  necessary  to  effectively  apply 
EVM  and  the  conversion  from  previous  program  management 
parameters  into  EVM  structures  did  not  consume  a  great  deal 
of  personnel  or  financial  resources.  The  aggregate  mean 
for  this  question  was  4.167,  with  corresponding  government 
and  contractor  means  of  4.333  and  4.000. 


Table  2 .  Means  for  Usability  Category  and  Questions 


Combined 

Government 

Contractor 

Overall  Usability 

3.889 

4.083 

3.694 

Ease  of  implementation  of 

EVM  system 

4.167 

4.333 

4.000 

Understanding  of  EVM  system 
and  tracking  information 

4.167 

4.500 

3.833 

Respondents  also  felt  comfortable  with  their 
understanding  of  how  the  EVM  system  is  applied  and 
interpreted.  The  aggregate  mean  was  4.167,  the  government 
mean  was  4.500  and  the  contractor  mean  was  3.833.  Again 
there  is  a  disparity  between  the  government  and  contractor 
scores,  with  the  contractor  score  being  substantially 
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lower.  While  the  contractor  scores  are  still  high,  this  is 
a  potential  concern  and  training  may  help  bridge  the  gap. 

Comparison  with  Previous /Other  Current  Systems 

The  lowest  scores  of  the  questionnaire  dealt  with  the 
comparability  of  EVM  with  previous  systems  of  project 
management.  An  important  consideration  for  choosing  to 
introduce  a  new  project  management  system  is  that  is 
provides  additional  capabilities  and  easier  administration 
than  the  system  being  replaced.  The  aggregate  response 
level  for  the  comparability  questions  (Appendix  C)  equaled 
3.306,  with  the  government  mean  equaling  3.611  and  the 
contractors'  exactly  3.000.  This  is  an  indication  that  the 
contractors  may  feel  EVM  is  simply  the  incorporation  of  the 
previous  systems  under  a  new  name. 


Table  3 .  Means  for  Comparability  Category  and  Questions 


Combined 

Government 

Contractor 

Overall  Comparability 

3.306 

3.611 

3.000 

Previous  cost  and  schedule 
tracking  difficulty 

3.500 

3.667 

3.333 

Preference  of  EVM  over 
other  management  systems 

3.083 

3.500 

2.667 
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Respondents  indicated  that  previous  systems  had 
difficulties  affecting  their  adequacy  to  track  cost  and 
schedule  performance.  Granted  that  EVM  parameters  are  a 
subset  of  the  extensive  35  C/SCSC,  but  the  response  levels 
show  an  enhanced  capability  of  EVM  to  monitor  cost  and 
schedule  performance  over  C/SCSC  and  other  systems.  The 
mean  response  level  for  the  entire  sample  was  3.500,  with 
corresponding  means  for  the  government  (3.667)  and 
contractor  (3.333)  groupings. 

The  results  were  inconclusive  pertaining  to  the 
preference  for  EVM  over  other  program  management  systems. 
The  combined  mean  was  3.083,  with  a  government  mean  of 
3.500  and  a  contractor  mean  of  2.667.  The  disparity 
between  the  means  of  the  government  and  contractor 
community  needs  to  be  addressed.  The  government  must 
discern  whether  the  contractors  are  using  different  systems 
for  internal  program  management  or  whether  refinements  may 
need  to  be  made  to  the  EVM  system. 

Life  Cycle  AppliceJaility 

Respondents  generally  agreed  that  EVM  has  potential 
for  application  across  several  stages  of  the  project 
management  life  cycle.  The  questions  concerning  life  cycle 
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applicability  category  (Appendix  D)  had  an  aggregate 
response  level  of  3.514,  with  the  respective  government  and 
contractor  means  of  3.639  and  3.389.  While  the  scores  were 
indicative  of  broad  application  across  the  several 
different  stages  of  the  life  cycle,  there  was  an  indication 
that  EVM  is  more  useful  for  some  stages  of  the  software 
project  management  life  cycle  than  others. 


T^d^le  4.  Means  for  Life  Cycle  Category  and  Questions 


Combined 

Government 

Contractor 

Overall  Life  Cycle  Applicability 

3.514 

3.639 

3.389 

EVM  more  useful  for  some  stages 
of  project  life  cycle 

3.000 

3.000 

3.000 

Management  of  systems  concept/ 
requirements  analysis  phase 

4.000 

4.000 

4.000 

Management  of  the  software 
system  design/coding  phase 

4.000 

4.000 

4.000 

Management  of  the  operations  and 
maintenance  phase 

3.000 

3.500 

2.500 

The  management  of  the  software  system  design/coding 
phase  was  found  to  receive  the  highest  scores  for  the 
application  of  EVM.  This  is  important  information  because 
this  has  been  a  difficult  area  for  acquisition  officials  to 
monitor  and  control.  It  is  also  beneficial  because  it 
allows  the  contractors  to  more  accurately  portray  progress 
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on  software  projects.  The  response  level  was  4.000  for  the 
combined,  government  and  contractor  means. 

Both  government  and  contractor  respondents  indicated 
that  EVM  is  more  useful  for  some  phases  of  system 
acquisition  than  others.  The  response  level  again  was 
4.000  for  the  combined,  government  and  contractor  means. 

The  areas  considered  to  be  less  conducive  to  earned  value 
management  will  now  be  explored. 

The  systems  concept /requirements  analysis  phase  was 
one  of  the  areas  where  EVM  was  found  to  be  less  useful  for 
project  management.  This  may  be  due  to  the  exploratory 
nature  and  difficulty  in  quantifying  the  work  of  the  phase. 
The  mean  was  3.000  for  the  combined,  government  and 
contractor  means. 

The  operations  and  maintenance  phase  was  another  area 
found  less  conducive  to  the ‘application  of  the  EVMS.  This 
is  likely  due  to  the  fluctuation  in  the  requirements  that 
may  occur  in  the  duration  of  the  operations  phase.  There 
are  surges  and  changes  in  operational  concepts  that  may  be 
difficult  to  foresee  or  quantify  at  the  onset  of  a  contract 
administration.  The  maintenance  phase  will  fluctuate  along 
with  the  changes  in  operations,  and  this  could  be  difficult 
to  budget  and  manage  with  the  EVMS.  The  combined  mean  was 
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3.000,  with  a  government  mean  of  3.500  and  contractor  mean 
of  2.500.  The  difference  in  means  between  the  government 
and  contractor  may  be  explained  by  their  vantage  points  as 
consumer  and  provider. 
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V.  Conclusions  and  Recommendations 

Overview 

This  chapter  identifies  the  findings,  limitations  and 
recommendations  for  follow-on  research  for  this  study.  It 
will  discuss  the  findings  pertaining  to  insight,  usability, 
comparability  and  life  cycle  applicability  in  accordance 
with  the  Earned  Value  Management  System.  The  limitations 
of  the  study  and  the  structured  questionnaire  will  be 
addressed.  Possibilities  for  follow-on  research  will  also 
be  presented  and  offered. 

Findings 

Earned  Value  Management  allows  an  adequate  level  of 
insight  into  project  management  for  both  the  government  and 
contractor  communities .  Government  employees  perceive  a 
slightly  higher  level  of  insight  than  their  contractor 
counterparts.  This  is  explained  by  the  fact  that  the 
Government  uses  EVM  as  a  project  status  and  control  tool, 
while  the  contractors  use  EVM  as  a  project-reporting  tool. 
Greater  insight  into  software  projects  will  help 
acquisition  officials  prevent  these  projects  from  becoming 
cost  and  schedule  prohibitive. 

Earned  Value  Management  is  very  helpful  in  the 
tracking  of  cost  and  schedule  in  project  management.  EVM 
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also  enables  acquisition  officials  to  identify  potential 
cost  and  schedule  variances  earlier  to  either  prevent  them 
or  implement  contingency  plans  to  minimize  their  effect. 
More  attention  needs  to  be  placed  on  the  transformation 
from  the  cost  estimation  metrics  into  earned  value  measures 
to  prevent  information  loss.  Attention  also  needs  to  be 
directed  to  the  metrics,  which  were  indicated  to  be 
inadequate  for  accurate  tracking  using  EVM. 

Government  and  contractor  employees  are  satisfied  with 
the  usability  and  implementation  of  EVM  into  project 
management.  This  is  an  important  consideration,  since  the 
rejection  by  either  party  would  doom  the  new  project 
management  system  to  failure.  Both  parties  must  have  a 
good  understanding  of  how  to  administer  EVM  and  interpret 
its  information  in  order  for  the  new  philosophy  to  be 
effective  and  successful. 

The  implementation  of  EVM  was  not  indicated  to  be 
resource  prohibitive.  Neither  the  contractor  nor  the 
government  indicated  that  too  many  resources  were  expended 
with  the  integration  of  EVM  into  their  programs. 

Personnel's  training  has  been  sufficient,  and  the 
conversion  from  previous  program  management  into  EVM 
parameters  did  not  pose  extensive  problems. 
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Program  participants  are  comfortable  with  their 
understanding  of  how  the  EVM  system  is  applied  and 
interpreted.  The  disparity  between  the  government  and 
contractor  scores  is  a  concern.  Attention  needs  to  be 
given  to  address  contractor  concerns  or  training 
deficiencies  related  to  Earned  Value  Management  philosophy. 

A  preference  for  EVM  over  previous  program  management 
schemes  is  inconclusive.  Government  employees  indicate  a 
preference  for  EVM  over  C/SCSC  and  other  program  management 
schemes,  but  the  contractors  have  no  preference.  If  the 
contractor  community  does  not  discern  a  difference  and  an 
enhanced  capability,  other  than  its  ability  to  track  cost 
and  schedule  progress,  EVM  runs  the  risk  of  being  another 
comprehensive  system  that  burdens  the  contractors. 

EVM  is  applicable  across  several  stages  of  the  project 
management  life  cycle.  Although  EVM  is  applicable  across 
several  different  stages,  EVM  is  more  useful  for  some 
stages  of  the  software  project  management  life  cycle  than 
others.  This  indicates  that  EVM  is  simply  one  of  several 
management  schemes  needed  for  management  of  the  project 
throughout  the  entire  life  cycle. 

Management  of  the  software  system  design/coding  phase 
is  well  suited  for  the  application  of  EVM.  This  has  been 
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one  of  the  most  difficult  areas  for  acquisition  officials 
to  monitor  and  control.  This  will  allow  more  accurate 
portrayal  of  progress  during  this  phase,  but  only  if  an 
accurate  understanding  of  the  coding  necessary  is 
represented  in  the  EVM  tracking  parameters. 

The  systems  concept/requirements  analysis  and  the 
operations/maintenance  phases  are  areas  where  EVM  is  less 
useful  for  project  management.  Attention  must  be  directed 
to  overcome  this  discrepancy.  Several  approaches  are 
available.  EVM  can  be  modified  to  be  better  applicable  to 
these  phases.  Other  program  management  schemes  can  be 
employed  for  these  phases  if  their  nature  is  not  conducive 
to  EVM  tracking.  If  possible,  training  can  be  accomplished 
to  show  how  EVM  is  usable  in  these  phases. 

Research  Reconmenda'bions 

The  following  recommendations  can  be  made  for  follow- 
on  research  from  this  study: 

a.  A  study  focusing  on  the  different  levels  of  management 
(senior,  middle  and  lower)  and  their  perceptions  of 
the  applicability  of  EVM  would  determine  if  gaps  exist 
between  government  and/or  contractor  management 
echelons . 
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b.  A  detailed  investigation  of  different  types  of 
acquisition  projects  (hardware  development,  software 
development,  COTS  integration,  operations  and 
maintenance)  would  determine  distinctions  in  the 
applicability  of  EVM. 

c.  A  case  study  showing  a  comparison  of  the  success  of 
EVM  and  C/SCSC  in  acquiring  similar  systems  would 
provide  a  comparison  of  the  their  ability  in  meeting 
cost,  schedule  and  performance  requirements. 

Conclusion 

Earned  Value  Management  is  a  program  management  tool 
that  is  easily  integrated  and  applied  by  the  government 
acquisition  community.  EVM  gives  government  officials  much 
greater  insight  into  the  cost  and  schedule  aspects  of  an 
acquisition  by  tracking  these  parameters  against  the  amount 
of  work  scheduled  to  be  completed  at  a  given  time.  EVM 
allows  better  forecasting  of  cost  and  schedule  variances 
and  planning  for  their  resolution.  The  applicability  of 
EVM  is  more  useful  for  some  phases  of  the  software 
acquisition  life  cycle  than  others,  and  program  management 
officials  must  find  alternatives  or  modify  the  EVMS  to 
account  for  peculiarities  of  the  different  phases. 
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Appendix  A:  Insight  Questions 


1)  EVM  provides  adequate  insight  into  cost  and  schedule 
performance . 

2)  EVM  allows  adequate  forecasting  of  cost  and  schedule 
variances . 

3)  EVM  better  allows  my  project  to  be  completed  within  cost 
and  schedule. 

4)  Adequate  tools  exist  to  translate  cost  estimation 
parameters  to  EVM. 

5)  Project  management  information  gets  lost  in  the 
transition  from  cost  estimation  to  EVM. 

6)  Current  software  metrics  are  adequate  for  tracking 
progress  using  EVM. 

7)  Software  defects  and  modifications  can  be  tracked  using 
EVM . 

8)  EVM  is  not  precise  enough  to  give  the  level  of  detail 
necessary  for  program  management. 

9)  EVM  allows  adequate  representation  of  contractor 
performance . 

10)  The  program  management  approach  used  by  the  contractor 
captures  the  data  needed  for  EVM  reporting  to  the 
government. 
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Appendix  B:  Usadaility  Questions 


1)  The  use  of  EVM  makes  program  management  easier. 

2)  The  use  of  EVM  is  not  worth  its  time  and  dollar  expense. 

3)  Software  projects  can  be  effectively  managed  using  EVM. 

4)  Hardware  projects  can  be  effectively  managed  using  EVM. 

5) 1  have  a  good  understanding  of  how  the  EVM  system 
actually  works. 

6)  EVM  shifts  the  government  cost  and  schedule  tracking 
burden  to  the  contractor. 
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Appendix  C;  Comparability  Questions 


1)  Previous  program  management  systems  were  adequate  for 
tracking  cost  and  schedule  performance. 

2)  The  term  "Earned  Value  Management"  is  a  buzzword  for  the 
same  old  system. 

3)  Better  program  management  systems  exist  that  EVM. 
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Appendix  D;  Life  Cycle  Applicability  Questions 


1)  Management  of  the  system  concept/requirements  analysis 
phase  is  enhanced  using  EVM. 

2)  Management  of  the  planning  phase  is  enhanced  using  EVM. 

3)  Management  of  the  software  system  design/coding  phase  is 
enhanced  using  EVM. 

4)  Management  of  the  system  implementation/testing  phase  is 
enhanced  using  EVM. 

5)  Management  of  the  operations  and  maintenance  phase  is 
enhanced  using  EVM. 

6)  EVM  is  more  useful  for  some  phases  of  the  system 
acquisition  than  others. 
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Appendix  E;  Structured  Questionnaire 


Please  use  the  following  scale  to  respond  to  the 
statements : 


Strongly 

Disagree 

Neither  agree 

Agree 

Strongly 

disagree 

or  disagree 

agree 

(1) 

(2) 

(3) 

(4) 

(5) 

1)  Earned  Value  Management  provides  adequate  insight 

into  cost  and  schedule  performance.  12345 

Comments : 


2)  Earned  Value  Management  allows  adequate  forecasting 

of  cost  and  schedule  variances.  12345 

Comments : 

3)  The  use  of  Earned  Value  Management  makes  program 

management  easier.  12345 

Comments : 

4)  The  use  of  Earned  Value  Management  is  not  worth 

its  time  and  dollar  expense.  12345 

Comments : 


5)  Previous  program  management  systems  were  adequate  for 

tracking  cost  and  schedule  performance.  12345 

Comments  (Which  systems?) : 

6)  The  term  "Earned  Value  Management”  is  a  buzzword 

for  the  same  old  system.  12345 

Comments : 
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7)  Earned  Value  Management  better  allows  my  project  to  be 
completed  within  cost  and  schedule.  12345 

Comments: 


8)  Better  program  management  systems  exist  than 

Earned  Value  Management.  12345 

Comments: 


9)  Management  of  the  system  concept /requirements 
analysis  phase  is  enhanced  using  Earned  Value 
Management .  12345 

Comments : 


10)  Management  of  the  planning  phase  is  enhanced 

using  Earned  Value  Management.  12345 

Comments : 


11)  Management  of  the  system  design/coding  phase  is 

enhanced  using  Earned  Value  Management.  12345 

Comments : 


12)  Management  of  the  system  implementation/testing 
phase  is  enhanced  using  Earned  Value 

Management .  12345 

Comments : 


13)  Management  of  the  operations  and  maintenance 
phase  is  enhanced  using  Earned  Value 

Management.  12345 

Comments : 
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14) 


Software  projects  can  be  effectively  managed 

using  Earned  Value  Management.  1  2  34  5 

Comments : 


15)  Hardware  projects  can  be  effectively  managed  using 

Earned  Value  Management.  12345 

Comments : 


16)  Adequate  tools  exist  to  translate  cost  estimation 
parameters  to  Earned  Value  Management.  12345 

Comments  (Which  tools?) : 

17)  Project  management  information  gets  lost 
in  the  transition  from  cost  estimation 

to  Earned  Value  Management.  12345 

Comments : 


18)  Current  software  metrics  are  adequate  for  traclcing 

progress  using  Earned  Value  Management.  12345 

Comments  (Which  metrics?) : 


19)  Software  defects  and  modifications  can  be  tracked 

using  Earned  Value  Management.  12345 

Comments: 


20)  Earned  Value  Management  is  more  useful  for  some  phases 
of  systems  acquisition  than  others.  12345 

Comments  (Which  and  why?) : 


21)  I  have  a  good  understanding  of  how  the  Earned  Value 

Management  system  actually  works .  12345 

Comments: 


22)  Earned  Value  Management  is  not  precise  enough 
to  give  the  level  of  detail  necessary  for 
program  management.  12345 

Comments: 


23)  Earned  Value  Management  shifts  the  government 
cost  and  schedule  tracking  burden  to  the 
contractor.  12345 

Comments : 


24)  Earned  Value  Management  allows  adequate  representation 
of  contractor  performance.  12345 

Comments: 


25)  The  program  management  approach  used  by  the  contractor 
captures  the  data  needed  for  Earned  Value  Management 
reporting  to  the  government.  12345 

Comments : 


26)  What  metrics  does  your  program  office  use  to  assess 

the  value  a  contractor  earns  within  a  given  reporting 
period?  Please  list  and  explain. 
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